Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sourcegraph: multi-tenant Zoekt #859

Merged
merged 13 commits into from
Nov 19, 2024
Merged

sourcegraph: multi-tenant Zoekt #859

merged 13 commits into from
Nov 19, 2024

Conversation

stefanhengl
Copy link
Member

@stefanhengl stefanhengl commented Nov 14, 2024

This updates webserver and sourcegraph-indexserver to support multi-tenancy.

The change is behind an ENV feature-flag.

Key changes:

  • tenant ID is now part of the index (repo metadata)
  • GRPC: IndexOption and Repository have a new field TenantId
  • If multi-tenancy is enabled, webserver checks if tenant in context matches the tenant id in the shard
  • zoekt-git-index has a new parameter "-shard_prefix ". If set, the value will be used instead of repository name as prefix for the name of the shard. For Sourcegraph we use "<tenant id>_<repository id>" as prefix if multi-tenancy is enabled (see screenshot).

Assumptions:

  • All calls to Sourcegraph are privileged

Test plan:

  • New tests
  • Ran this together with Sourcegraph (with and without MT enabled)

This is how the new Sourcegraph naming scheme looks like on disk:
image

eval.go Show resolved Hide resolved
Copy link
Member

@keegancsmith keegancsmith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels so much better than the other PR right? dig it.

build/builder.go Outdated Show resolved Hide resolved
cmd/zoekt-sourcegraph-indexserver/index.go Outdated Show resolved Hide resolved
cmd/zoekt-sourcegraph-indexserver/index.go Outdated Show resolved Hide resolved
cmd/zoekt-sourcegraph-indexserver/index.go Outdated Show resolved Hide resolved
eval.go Show resolved Hide resolved
internal/tenant/query.go Show resolved Hide resolved
- prefix flag instead of bool
- padded ids
- include tenant in id
- add tenant tests
Copy link
Contributor

Fuzz test failed on commit b93cb3c. To troubleshoot locally, use the GitHub CLI to download the seed corpus with

gh run download 11855426986 -n testdata

@stefanhengl stefanhengl changed the title sourcegraph: multi-tenant Zoekt v2 sourcegraph: multi-tenant Zoekt Nov 15, 2024
@stefanhengl stefanhengl marked this pull request as ready for review November 15, 2024 12:32
build/builder.go Outdated Show resolved Hide resolved
build/builder.go Outdated Show resolved Hide resolved
@@ -153,6 +158,88 @@ func TestBasic(t *testing.T) {
})
}

func TestBasicTenant(t *testing.T) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: should this test live inside of the tenant package?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would create circular imports. Why do you want to move it? To not spill too much "tenant" into the codebase? We could put it in a separate package inside internal/tenant/

cmd/zoekt-sourcegraph-indexserver/index.go Outdated Show resolved Hide resolved
cmd/zoekt-sourcegraph-indexserver/index.go Outdated Show resolved Hide resolved
@@ -592,6 +601,7 @@ func (s *Server) Index(args *indexArgs) (state indexState, err error) {
case build.IndexStateMeta:
log.Printf("updating index.meta %s", args.String())

// TODO(stefan) handle mergeMeta for tenant id.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:O also need exploding then to understand the new naming scheme?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This here is a bit different. We should probably trigger a reindex if the tenant id changes, but I didn't want to include it now because that would trigger a reindex for everyone.

For explode, I updated the naming gated by tenant enforcement. I will file a follow-up to make that nicer. It's obviously too brittle.

cmd/zoekt-sourcegraph-indexserver/main.go Outdated Show resolved Hide resolved
stefanhengl and others added 3 commits November 19, 2024 08:48
Co-authored-by: Keegan Carruthers-Smith <[email protected]>
Co-authored-by: Keegan Carruthers-Smith <[email protected]>
@stefanhengl stefanhengl merged commit c52a9cd into main Nov 19, 2024
9 checks passed
@stefanhengl stefanhengl deleted the sh/multitenant-zoekt-2 branch November 19, 2024 11:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants